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SUMMARY 


In this study, we aim to characterize how students enrolled in a test 
preparation product, ACT® Online Test Prep (AOP), use that 
product. Following Geiser (2012), Masyn (2013), and Morin (2016), 
we use latent profile analysis (LPA) to identify students’ 
engagement profiles using four engagement measures: activity, 
time, practice ACT score, and percent correct. Five engagement 
profiles were identified: Low-Usage/Low-Performance (24%), Low- 
Usage/High-Performance (7%), Low-Usage/Moderate-Performance 
(32%), Moderate-Usage/Low-Performance (17%), and High-Usage/ 
Moderate-Performance (20%). Moreover, female students were 
less likely to have a Low-Usage/High-Performance profile. 

The heterogeneity of engagement profiles is discussed in 

relation to usage and performance as well as potential future 
research directions. 


SO WHAT? 


This study is a first step towards understanding how students engage 
with online programs for preparing for college entrance exams. 
Understanding variation in test preparation usage can help inform 
product improvement efforts and guide new users to get the most from 
their preparation. 


NOW WHAT? 


Future research should include additional measures of engagement 
for identifying engagement profiles, test whether additional student 
characteristics are predictive of engagement profile, and examine 
differences in ACT score gains across engagement profiles. 
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Examining Profiles of Engagement with 


ACT Online Prep 


Abeer A. Alamri, PhD, and Edgar I. Sanchez, PhD 


Introduction 


Tests traditionally used for college admissions 
(e.g., ACT® and SAT) can play an important role 
in helping students attain their academic goals. 
Students are motivated to maximize their score 
in order to meet admissions requirements, 
maximize scholarship opportunities, and avoid 
remediation. In 2017, just over two million high 
school graduates, approximately 60% of high 
school graduates, took the ACT with an average 
ACT Composite score of 21.0 (ACT, 201 7a; 
ACT, 2017b). Of those students, 27% met all 
four ACT College Readiness Benchmarks. 


Students can use many different tools to prepare 
for the ACT test, including resources from ACT, 
which provide different types of supports and 
content review. Within the test preparation 
literature, research has demonstrated the 
positive impact of test preparation on test scores 
(e.g., Briggs, 2009; Messick & Jungeblut, 1981; 
Moore, Sanchez, & San Pedro, 2018; Sanchez 
& Harnisher, 2018). Some research has studied 
motivational factors that may affect student 
preparation (e.g., Appelrouth, Zabrucky, & 
Moore, 2015), while other research has 
investigated the causal evidence of the impact of 
test preparation on scores (Moore et al., 2018). 


However, different usage patterns of test 
preparation products are not widely studied. 
Therefore, we seek to identify usage profiles for 
ACT Online Prep (AOP). In this manuscript, we 
elaborate on key features of AOP, present the 
empirical data, discuss the results, and 
summarize recommendations for future 
researchers. 


ACT! offers a variety of test preparation materials 
that feature materials from previous tests 
including: the Official ACT Prep Guide, ACT® 
Academy™, ACT Online Prep (AOP), and a 
collaboration with Kaplan Test Prep to offer an 
interactive virtual classroom and library or videos 
called ACT® Rapid Review. In this paper, we 
limited our examination of test preparation to AOP. 


AOP is a subscription-based online service with 
access to a dynamic, interactive test preparation 
course designed by ACT. It consists of six 
components that target skills in the four subjects 
(English, mathematics, reading, and science): 
practice sessions, instructional lessons, ACT 
practice tests, discussion boards, educational 
games, and flashcards. The program offers a 
personalized learning path, and students can 
choose from a variety of activities with 
comprehensive content review. Predicted scores 
and feedback are provided, and users can reset 
the activities, including tests. 


Practice Sessions 


Practice sessions consist of diagnostic and 
practice questions. This feature consists of more 
than 2,400 practice questions (items) that cover 
the four ACT subjects. The practice items 
facilitate learning through scaffolding with 
immediate feedback. 


Instructional Lessons 


Instructional lessons are extensive reviews of 
content covered within the four ACT subject tests. 
For example, the math lessons prepare students 
for math topics such as statistics and probability, 
Algebra, functions, number and quantity, and 
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Geometry. At the end of each lesson, students 
are asked to report their confidence level in that 
topic (low, medium, high). 


ACT Practice Tests 


When students take an ACT practice test, they 
may pick a short or long form. The timed short 
ACT test is a limited number of test items in one 
of the four ACT subjects; once completed, 
students are provided with a predicted ACT 
subject test score range. The long form practice 
test consists of timed, retired ACT tests that 
mimic the official four-subject ACT test. At the 
end, students receive estimated ACT scores for 
each of the four individual subject tests as well as 
an estimated Composite score. Students are able 
to reset and retake both types of practice tests. 


Discussion Boards, Educational 
Games, and Flashcards 


These AOP components focus on peer/group 
interaction via a discussion board where 
students discuss test preparation or ask 
questions in order to learn from each other. 
Individual learning is also supported using 
educational games and flashcards, targeting 
specific concepts covered on the ACT. 


Background: Test 
Preparation Efficacy 
Research 


Students come from different backgrounds and 
are exposed to different educational settings and 
instruction; therefore, personalized learning 
plans are an effective way to provide 
individualized opportunities (NACAC, 2015). The 
same holds true for test preparation strategies, 
which can provide students with individualized 
plans to improve their mastery of tested 
knowledge and skills. Instead of following a 
common strategy, students may use many types 
of strategies to arrive at a final answer when 
taking standardized tests (Baleghizadeh & 
Yousefian, 2012). Further, understanding the 
construction of tests is a valid preparation 


approach (Baleghizadeh & Yousefian, 2012). 
Worthwhile preparation activities could include 
practicing different types of questions under time 
restraints, utilization of vocabulary knowledge, 
and ways to eliminate wrong answer choices. 


Several factors that influence the effectiveness 
of test preparation, such as baseline 
achievement level, school attendance, and 
preparation participation, have been examined. 
For example, allotted time and type of test 
preparation were found to have positive 
influences on test scores. Messick and 
Jungeblut (1981) studied the impact of time and 
methods of test preparation on improvement in 
SAT score by examining previously published 
studies. They emphasized that both amount of 
time and duration of test preparation positively 
impacted scores. They further noted that the 
effect of time was confounded with other 
aspects of test preparation (i.e., increasing 
curriculum emphases on content knowledge and 
skill development). 


Appelrouth, Zabrucky, and Moore (2015) noted 
that factors such as gender, school type, 
homework completion, time allotted, and tutoring 
type (i.e., individual versus groups) could impact 
the effect test preparation has on SAT scores. 
They collected data from 1,933 junior and senior 
students who participated in SAT test 
preparation programs in three large metropolitan 
areas. They also collected demographic 
information, prior SAT score, attendance, 
preparation participation, and post SAT score. 
Time allotted on practice tests and type of 
preparation (e.g., individual versus group 
tutoring) were significant factors in predicting 
score gain. They stated that each individual 
preparation hour contributed to a 2.34-point gain 
on the SAT (0.07 SD units). Both time and 
type of test preparation were positively related to 
students’ test scores. 


Several studies have been conducted which 
examine the effect of ACT test preparation. 
Sanchez and Harnisher (2018) investigated the 
effectiveness of test preparation on ACT score 
gains for students that had taken the test more 
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than once by comparing scores for a treatment 
group who enrolled in test preparation and a 
control group who did not. Using a quasi- 
experimental design, they found that the 
treatment group had higher retest scores (0.13 
SD units) than their control group counterparts. 
They also found that this increase was greatest 
for low-income students. 


Moore, Sanchez, and San Pedro (2018) 
examined the impact of 10 different ACT 
preparation tools on ACT retest scores. The 
findings indicated that students who participated 
in test preparation had higher ACT Composite 
scores compared to students who did not. The 
adjusted ACT Composite mean was 24.33 for the 
test preparation group versus 23.63 for the non- 
test preparation group (0.14 SD units). Moreover, 
the adjusted mean score varied based on time 
spent on test preparation (e.g., adjusted mean 
score of 24.91 when student studied 11 hours or 
more with a private tutor or consultant). 


The Current Study 


Purpose and Research 
Questions 


This exploratory study seeks to identify distinct 
profiles of test preparation usage; this approach 
appears to be unique based on our survey of the 
test preparation literature. This study examines 
the frequently occurring combinations of four 
engagement factors among students who 
enrolled in AOP. We apply latent profile analysis 
(LPA) techniques to identify profiles of AOP 
usage and performance. Once specific profiles 
are defined, a secondary purpose is to test for 
gender differences in profile frequency. While 
there are many variables of interest besides 
gender that one could examine, we chose to 
examine gender because it is a key 
demographic characteristic as well as to 
illustrate the methodology. Specifically, we seek 
to answer these questions: Is there a profile 
structure that adequately represents the 
heterogeneity of AOP usage (i.e., are there 
patterns of engagement)? If so, what are they 
and what is their prevalence? Are male and 


female students equally represented across 
different profiles? 


Addressing these questions will provide insight 
into how test preparation strategies can be 
refined to address the needs of different user 
profiles. We hypothesize that two or more AOP 
student profiles will emerge. We also expect to 
observe profile differences in usage of AOP 
components and levels of performance. 


Methods 
Study Population and Design 


AOP data were collected from December 2015 
to June 2018. We collected data from practice 
sessions taken, practice tests taken, lessons 
viewed, and practice and test resets.4 We 
excluded cases when there were no records for 
the students (the student did not use that 
particular AOP component), as an attempt to 
have a full set of valid values across all 
variables, leaving 9,017 students for the 
analysis. All data descriptive analyses are 
conducted using SAS Enterprise Guide 7.1. 


Student race/ethnicity, gender, and family 
income were retrieved from ACT test records 
and are included as covariates. A summary 
table of the sample characteristics is presented 
in Table 1. Just over half of the students did not 
provide their family income, and one-third of the 
students did not provide their gender. Among 
those who provided data, there were equal 
numbers of male and female students and most 
students were White. We did not have access to 
data indicating student’s use of additional test 
preparation material beyond AOP. 
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Table 1. Study Sample Characteristics (N= 9,017) 


Characteristic N Percent 
Family Income 


Low-Income 317 4 
Middle-Income 1,485 16 
High-Income 2,484 28 
Missing 4,731 52 
Gender 
Male 3,069 34 
Female 2,958 33 
Missing 2,990 33 
Race/Ethnicity 
White 3,785 42 
African-American 18 0 
Hispanic/Latino 475 5 
Asian 776 9 
Other 973 11 
Missing 2,990 33 


Note. Percentages may not sum to 100% due to 
rounding. 


Measures. Data for the 9,017 students were 
collected on usage and performance within 
AOP. We used data relating to practice 
sessions, practice tests, lessons, and number of 
practice and test resets. In this analysis, we 
elected to focus on overall usage of AOP as 
opposed to subject-specific usage of AOP. As a 
result, we aggregated across subject-specific 
sections of AOP and used estimated ACT 
Composite score from long form practice tests.® 
While many different types of AOP data 
elements were available, 10 aggregated data 
elements were utilized in this study, including: 


1. Total number of practice sessions taken 


2. Total number of long-form practice tests 
taken 


3. Total number of lessons taken 
4. Total number of test section system resets 


5. Total number of practice section system 
resets 


6. Total time duration for practice sessions 
taken 


7. Total time duration taking practice ACT tests 
(long and short forms) 


8. Average predicted ACT Composite score 
from long-form practice tests taken®’ 


9. Average percent of questions answered 
correctly within practice sessions 


10. Average percent of questions answered 
correctly on practice ACT tests (long and 
short forms) 


We constructed four variables from these 10 
data elements. The four summary measures 
were activity, time, practice ACT score, and 
percent correct (see Table 2). Activity is 
measured as the mean of four variables. 
Because the four variables are on different 
scales with different variances and distributions, 
these values were standardized and then 
averaged to ensure that the scale of the 
variables did not have an undue influence on the 
overall measure. 
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Table 2. Description of the AOP Summary Measures 


AOP Summary Measures Description 


The arithmetic mean of the standardized total number of practice 


sessions taken across subjects, total number of long-form 


Penny practice tests taken, total number of lessons taken across 
Usage subjects, and total number of test section system resets 
Sum of the total time duration for practice sessions taken across 
Time subjects and total time duration taking practice ACT tests (long 


and short forms; time spent on lessons was not available) 


Practice ACT score 
ests 


Performance 
Percent correct 


Data Analysis Procedure 


To answer the main research question, we used 
the Mplus option “TYPE=MIXTURE?” to fit an 
LPA model starting with a parsimonious one- 
profile model without covariates and then 
increased the number of latent profiles to two, 
three, four, and five profiles.® For iterative 
testing, we used different sets of random start 
values (i.e., 500, 50) to avoid local maxima in 
determining the likelinood parameters with the 
Robust Maximum Likelihood (MLR) estimator. 
Then, overall model evaluation was based on fit 
statistics alongside a careful inspection of the 
nature of the groups and the proportion of 
students in each profile (i.e., model results, 
estimated profile size, entropy value, 
classification reliability, and class-conditional 
parameters). To compare across all profile 
models, the Log likelihood (LogL) and the 
Information Criteria (IC), Akaike Information 
Criterion (AIC; Akaike, 1973), Bayesian 
Information Criterion (BIC; Schwarz, 1978), and 
Sample-Size Adjusted BIC (SSABIC; Sclove, 
1987) were used. 


For comparisons of neighboring profile models 
(e.g., two-profile versus three-profile), we used 
both the Lo-Mendell-Rubin LMR (LRT) and 
Bootstrap LR Difference Test (BLRT) with 500 
replications with p < .05 as indications of model 
significance. Based on the model selection 


Average predicted ACT Composite score from long form practice 


The average percent of questions answered correctly within 
practice sessions and on practice ACT tests (long and short 
forms) 


process and class interpretability, we selected a 
suitable number of profiles. We named and 
interpreted the profiles based on the four 
engagement variables (activity, time, practice 
ACT score, and percent correct). 


Finally, to test whether males and females were 
equally represented within profiles, we assigned 
each individual to the latent profile for which her 
or his assignment probability was greatest, and 
then we used the “R3STEP” option in Mplus and 
added gender (female=1 and male=0) as an 
auxiliary latent class predictor to the LPA model. 
Of note, only AOP users with known gender are 
included (N=6,027, missing=2,999). Using one 
of the profiles as a reference group, positive 
values indicate that female students are more 
likely to be in the corresponding latent profile 
relative to male students (i.e. demonstrate this 
pattern of usage). All statistical tests were two- 
tailed, and significance was determined at the 
0.05 level using the Mplus statistical package 
version 7.4 (Muthén & Muthén, 1998-2015). 


ACT Research & Policy | ACT Research Report | R1755 


Results 


We aimed to identify subpopulations of students 
based on their engagement with AOP. Data 
were screened for outliers,° and the distribution 
for each measure was observed. Table 3 
presents the descriptive statistics for each 
measure to help contextualize sample 
characteristics. We used normal probability plots 
and histograms to check for normality. Practice 
ACT score and percent correct were 
approximately normally distributed, whereas the 
activity data had a slight departure from 
normality. The probability plot shows a linear 
relationship with minor deviations. Skewness of 
the four measures ranged from -.60 to 1.64, with 
kurtosis ranging from -0.21 to 2.50. 


Correlations between the four AOP measures 
were positive: activity and time (r = .56), 
activity and practice ACT score (r =.11), 
activity and percent correct (r =.15), time and 


practice ACT score (r =.12), time and percent 
correct (r =.11), and practice ACT score and 
percent correct (r =.67). 


AOP Latent Profile Analysis 


The validity of inferences made from LPA is 
dependent upon having good model fit and 
meaningful interpretation of profiles. In this 
study, we found that the five-profile model was 
most appropriate. Relative to the one- through 
four-profile models, the five-profile model 
exhibited lowest LogL, AIC, BIC, and SSABIC 
values, high reliability and entropy, as well as 
non-significant LMR and BLRT values. In the 
five-profile model, the entropy is .81, suggesting 
clear profile separation as well as high values of 
precision (>.86) of correct profile assignment 
probabilities (.88, .91, .86, .89, and .92) for each 
profile. Table 4 provides a summary of the fit 
statistics for possible latent profile structures to 
each of the five extracted profiles. 


Table 3. Descriptive Statistics of AOP Engagement Summary Measures (N= 9,017) 


AOP Engagement Standard 

Measure Deviation Skewness Kurtosis 
Activity 0.27 0.75 0.97 0.45 
Time 7.99 2.45 1.64 2:50 
Practice ACT score 23.44 6.88 -0.60 -0.21 


Percent correct 48.52 17.79 0.43 -0.10 
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Table 4. Summary of the Fit Statistics for Possible Latent Profile Structures (N= 9,017) 


BLRT k-1 LMR k-1 


Profile Profile 
Versus k Versus k 
Profile Size Entropy Reliability SyAV-N =] | Os Profile Profile 
One-profile 100 - - 8 -51178.275 102372.550 102429.405 102403.982 - - 
55.8 93 
Two-profile 1 17 -45732.911 91499.821 91620.638 91566.615  p=.000 p =.000 
44.2 94 
25.6 .88 
Three-profile 39.2 16 94 26 -44006.356 88064.712 88249.490 88166.866  p=.000 p =.000 
35.2 87 
20.2 87 
31.3 85 
Four-profile ae 19 as 30 -42040.697 84151.395 84400.135 84288.911 p =.000 p =.000 
27.9 93 
24.1 88 
07.2 91 
Five-profile S17 81 86 44 -41327.162 82742.325 83055.027 82915.202 p=.000 p =.000 
16.8 89 
20.2 92 


Note. Reliability= reliability of classification; NP= number of free parameters; LogL= Log likelihood; AlC= Akaike Information Criterion; BIC= Bayesian Information Criterion, 
SAABIC= Sample-Size Adjusted BIC; BLRT= Bootstrap Likelihood Test; LMR= Lo-Mendel Rubin Test; k= profile number. 
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We denoted students’ engagement as low- 
usage/low-performance, low-usage/high- 
performance, low-usage/moderate-performance, 
moderate-usage/low-performance, and high- 
usage/moderate-performance. Figure 1 
demonstrates how the five-profile model helps 
illustrate the differences in engagement 
predictors (e.g., percent correct and time). 
Similar plots, not shown due to redundancy, 
demonstrate the differentiation of engagement 
by profiles on other predictors. 


Interpretation of the Five 
Profiles of AOP Engagement 


Figure 2 shows the standardized means for 
each profile across the four AOP engagement 
variables (i.e. activity, time, practice ACT score, 
and percent correct in each of the five latent 
profiles). High standardized means indicate 
higher values of that variable in the respective 
profile while lower standardized means indicate 
lower values of that variable. For example, 
students in the low-usage/high performance 
profile had a below average number of activities 
and hours on practice items and tests but 
above average practice ACT scores and 
percent correct. 


Figure 1. Time and Percent Correct Scatterplot for the Five-Profile Model Predictors 
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Note. Blue: Low-Usage/Low-Performance, Yellow: Low-Usage/High-Performance, Orange: Low-Usage/Moderate- 
Performance, Green: Moderate-Usage/Low-Performance, Light Blue: High-Usage/Moderate-Performance 
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Figure 2. Five-Profile Model with Standardized Means of AOP Engagement Predictors 
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In addition to the profile size (Table 4), we used 
the 95% confident interval (Cl) of means and 
standard deviations predictor values (Table 5) to 
better understand the engagement profiles. In 
general, if the AOP engagement measure mean 
is less than zero, we treated it as low usage or 
low performance, and when it is greater than 
zero, we labeled it as moderate or high. Low- 
usage/low-performance students (N= 2,176) had 
a low number of activities, invested little time 
using AOP, and had the lowest ACT practice 
test score as well as the lowest percent correct 
on test and practice items. Low-usage/high- 
performance students (N=625) had a low 
number of activities and invested little time using 
AOP but had the highest practice ACT scores 
and percent correct on test and practice items. 


In contrast, low-usage/moderate-performance 
students (N=2,856), the profile with the largest 
number of students, had a low number of 
activities and invested little time using AOP but 
had moderate practice ACT scores and percent 
correct on test and practice items. Moderate- 
usage/low-performance (N=1,511) students had 
a good number of AOP activities and hours 
using practice items and tests but low practice 
ACT scores and percent correct on test and 
practice items. Finally, high-usage/moderate- 
performance students (N=1,821) had the highest 
number of AOP activities and hours using 
practice items and tests but had moderate 
practice ACT scores and percent correct on test 
and practice items. 
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Table 5. 95% Confidence Intervals of the AOP Engagement Summary Measures by Profile (N= 9,017) 


Activity led (ea (ex: WAN Of bs loxo) x=) Percent Correct 
Profile and 
Parameter Est U 2.5% L 2.5% Est U 2.5% L 2.5% Est U 2.5% L 2.5% Est 
Low-usage/ M -0.98 -0.96 -0.94 -0.73 -0.72 -0.71 -0.44 -0.35 -0.27 -0.42 -0.33 -0.24 
low- 
performance SD 0.17 0.19 0.20 0.17 0.18 0.19 0.92 0.96 0.99 0.85 0.90 0.95 
Low-usage/ M -0.46 -0.35 -0.23 -0.70 -0.68 -0.65 0.89 0.97 1.04 2.00 2.07 2.15 
low- 
performance SD 0.88 0.97 EOS 0.18 0.20 0.22 0.36 0.39 0.42 0.33 037 0.40 
Low-usage/ M -0.11 -0.07 0.03 -0.24 -0.21 -0.17 0.34 0.39 0.43 -0.08 -0.03 0.01 
moderate- 
performance SD 0.49 0.51 0.52 0.34 0.36 0.38 0.62 0.65 0.68 0.54 0.57 0.59 
Moderate- M 0.20 Org 0.37 0.06 0.17 0.28 -1.35 -1.27 -1.19 -1.07 -1.01 -0.95 
usage/low- 
performance SD 0.91 0.96 1.00 1.01 1.09 1.16 0.68 0.70 ONS 0.51 0.54 0.56 
High-Usage/ M 11 OM 1.14 1.20 1.20 1.29 1.37 0.48 0.53 0.58 0.50 0.55 0.60 
moderate- 
Performance SD OFZ O15 0.98 1.06 1.09 1.12 OFog 0.62 0.65 0.57 0.60 0.62 


Note. Est = Estimated parameter; M= Mean; SD= Standard Deviation; L 2.5%= lower 2.5%; U 2.5%= Upper 2.5% 
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Figure 3. Profile Class Proportions for the Five-Profile Model 
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Note. Blue: Low-Usage/Low-Performance, Yellow: Low-Usage/High-Performance, Orange: Low-Usage/Moderate- 
Performance, Green: Moderate-Usage/Low-Performance, Light Blue: High-Usage/Moderate-Performance 


Examining the estimated profile memberships moderate-usage/low-performance, and light blue 
and associated probabilities reveals that within for high-usage/moderate-performance. 

each profile, each AOP user has a high 

probability of being in a specific profile and a For example, within low-usage/low-performance 
small to negligible probability of being in the profile memberships, most students are 

other profiles. The classification probabilities for classified as low-usage/low-performance users 
the most likely profile membership are presented (86.3%) with 10.4% being classified as low- 

in Figure 3 where each profile bar is depicted by usage/moderate-performance, 2.3 % as 

five colors: blue for low-usage/low-performance, moderate-usage/low-performance, and less than 
yellow for low-usage/high-performance, orange 2% as moderate-usage/low-performance and 


for low-usage/moderate-performance, green for high-usage/moderate-performance. 
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The classes do not have perfect membership 
because the model does not fit the data perfectly. 


Figure 4 shows the standardized mean of each 
AOP variable for the five profiles we identified. As 
shown, the standardized means for the four AOP 


measures (i.e., activity, time, practice ACT score, 
and percent correct) differ for each profile, which 
visually confirms the appropriateness of the five- 
profile model rather than the one-, two-, three-, or 
four-profile models. 


Figure 4. Contribution of Each AOP Engagement Predictor to the Five-Profile Model 
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Gender as a predictor of profile. We 
investigated the relationship between AOP 
engagement profile and gender. Since we have 
about 33% of the AOP users with no gender 
data, the Mplus default for listwise deletion is 
applied to the auxiliary variable, gender, in the 
analysis. Of the 9,017 AOP users, 2,999 were 
excluded, and a total of 6,027 AOP users are 
used in the gender analysis. The gender 
analysis provides the statistics to compare each 


Aelivs) 


Practice ACT score Percent correct 
= Low-Usage/High-Performance 


= Moderate-usage/Low-performance 


profile against a reference profile. The results 
showed that the female users are 
underrepresented in the low-usage/high- 
performance profile only. Using any other profile 
as a reference group, the significant effect of 
gender (female) for the low-usage/high- 
performance profile ranged between -.49 and - 
.69 (p<.001, odds ratio ranged between .61 and 
.00) and was not statistically significant for other 
profiles. The other four profiles showed female 
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overrepresentation against the low-usage/high- 
performance profile only if the low-usage/high- 
performance profile is used as a reference 
group. In other words, when compared to the 
low-usage/high-performance profile, the 
probability of females being in one of the other 
four profiles significantly increases. Although 
females are less likely to be in the low- 
usage/high-performance profile, the gender 
analysis results demonstrate that females were 
a heterogeneous group of AOP users that were 
broadly distributed across the other four profiles. 
While it is unclear why female students tend not 
to fall into the low-usage/high-performance 
profile, it would be worthwhile to explore what 
drives them to greater membership in other 
profiles. 


Discussion 


AOP is a learning tool designed to improve the 
knowledge and skills assessed on the ACT, 
which will ultimately help students succeed in 
college and in their careers. Understanding the 
patterns of engagement in test preparation 
resources is necessary for developing and 
improving test preparation tools and 
interventions. Within the context of test 
preparation for college entrance exams, prior 
studies have not widely examined patterns of 
engagement. Results of our study revealed five 
profiles that exemplify students’ engagement 
behaviors as low-usage/low-performance, low- 
usage/high-performance, low-usage/moderate- 
performance, moderate-usage/low-performance, 
and high-usage/moderate-performance. We 
found greater numbers of students in the low- 
usage/moderate-performance, the low- 
usage/low-performance, and the high- 


usage/moderate performance profiles. Moreover, 


female students were underrepresented in the 
low-usage/high-performance profile. A better 
understanding of the motivations and strategies 


of AOP usage would help us understand why this 


is the case. 


These student profiles may help developers 
target specific interventions and inform the 
degree to which interventions may be effective. 


13 


This information is particularly salient in the 
context of test preparation for college entrance 
exams since engagement patterns with test 
preparation products has not been studied. The 
results revealed three noteworthy findings. 


1. The identification of five engagement profiles 
of AOP engagement. The results provide 
greater clarity about factors that characterize 
student experiences. It allows further 
understanding of test preparation users and 
how their engagement behaviors, including 
usage and performance, could be associated 
with gender. 


Sanchez and Harnisher (2018) propose two 
types of students who may use test 
preparation: high achievers seeking exceptional 
scores and lower achievers seeking 
supplementary instruction. In our study, we 
identified low-usage/high-performance students 
who may be aligned with the high achievers 
proposed by Sanchez and Harnisher while 
adding two additional types: high- 
usage/moderate-performance students who are 
highly engaged with test preparation materials 
in terms of activity and time while they have a 
moderate performance level based on their 
practice ACT score and percent correct, and 
low-usage/low-performance students who 
make little use of test preparation and have 
lower achievement. While these results cannot 
provide insights into the motivations of and 
preparation strategies employed by students, it 
does raise questions about why students used 
AOP in the manner illustrated by their profile. 
Of particular interest are the differential 
strategies being employed by high- and low- 
usage students. It may be possible that low- 
usage students were not engaged with the 
product because they were not the purchasers 
of the program and failed to see the value in 
the offering. Additionally, high-usage students 
may have been highly motivated to increases 
their scores. 


2. Although females were less likely to be in the 


low-usage/high-performance profile, results 
demonstrated that female students were a 
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heterogeneous group of test preparation 
users that were broadly distributed across the 
other four profiles. 


3. It is important to consider students’ strategies 
of how to utilize the features of the program. 
It is of particular note that while low- 
usage/high-performance students had limited 
usage of AOP, they are among the highest 
achievers in terms of their practice ACT test 
scores and performance in practice sessions. 
It is possible that these students use their 
preparation time for practice items and tests 
rather than reviewing content because they 
feel they have mastery of the content. 
Alternatively, they may be higher-achieving 
students who do not see an advantage to 
using AOP. 


On the other hand, high-usage/moderate- 
performance students completed a high 
number of AOP activities, spent higher 
numbers of hours using practice items and 
tests, and had practice ACT score and 
percent correct around the average. Since 
these students are moderate achievers 
based on their practice test scores and 
percent correct, they need to be more 
efficient with their AOP time and engage in 
different types of activities that fulfil their 
academic needs. It is also possible that these 
are average achieving students that are using 
AOP to improve their ACT scores and 
therefore their postsecondary opportunities. 


Limitations 


There are a few limitations to this research worth 
discussing. First, while we made use of the AOP 
data we had available, there is a wealth of 
additional usage data that could be obtained 
from the AOP platform. Future research will 
need to examine this more detailed data to 
better understand how students are using AOP. 
Second, this exploratory study did not make use 
of students’ demographic information such as 
race/ethnicity, family income, coursework taken, 
high school GPA, or parent’s education. Each of 
these may serve to illuminate our understanding 


of why test preparation products are being used. 
In particular, they may help us understand why 
students engage with test preparation and what 
needs they are attempting to meet with their 
program use. 


Additionally, because our analysis approach 
requires non-zero usage measures, we only 
included students who engaged with AOP 
enough to have a non-zero value for each AOP 
usage variable. This means that the sample 
excludes students who did not engage with all of 
the AOP components studied. This reduces our 
ability to generalize to students who purchase 
but make poor use of the product but allows us 
to learn about students who purchase and make 
good use of the product. While this study looks 
at students who are using the product in a model 
consistent with intended use, it does not include 
other models of use. Finally, in this study we did 
not collect data on other forms of test 
preparation students may be using, and future 
studies should explicitly control for this effect. 


Practical Implications of the 
Study and Recommendations 
for Future Research 


The objective of this study was to better 
understand the different usage profiles of AOP 
users. Understanding how users engage with 
test preparation helps inform improvement 
efforts and can help guide how we advise new 
users to get the most from their preparation. If 
test preparatory intervention was implemented 
at a community level, for example at a school, 
this study suggests that about 20% of the 
students, on average, may have high usage 
coupled with moderate performance. However, 
to support this claim, students’ usage 
information and factors driving their preparation 
are needed, which can be gained through direct 
observation or think-aloud interviews. 


This study demonstrates the use of LPA and may 
serve as a useful approach for examining learning 
behavior patterns with other programs. To our 
knowledge, no research has systematically 
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investigated test preparation usage profiles by 
gender. Showing how this can be accomplished 
makes this process more transparent. In addition, 
tools like the Mplus mixture analysis that simplify 
this process are beneficial for researchers seeking 
to investigate latent profile properties associated 
with test preparation. 


Future research could expand on the current 
study in a number of ways. First, researchers 
may include additional measures of engagement 
when estimating user profiles. For example, 
more research is needed to investigate the AOP 
discussion board activity regarding peer 
influence on student engagement. Second, a 
study could investigate patterns of test 
preparation for only females or males and 
determine the most beneficial activities. In 
addition to gender, there are many other 
covariates that may be of interest in regard to 
test preparation. Third, regarding statistical 
models, since there is usually a general or 
second-order factor that affects the identification 
of individuals’ pattern (e.g., computer literacy or 
Internet reliability), using a bi-factor model or 
second-order model may enrich the profile 
results (Dantlgraber, Wetzel, Schutzenberger, 
Stieger, & Reips, 2016; Raykov, Marcoulides, 
Menold, & Harrison, 2019). 


Also, a multilevel LPA model may detect 
different patterns based on student, school, or 
community characteristics. Alternatively, future 
research could explore usage profiles across 
different test preparation programs to better 
understand how different usage strategies may 
be deployed across programs and why those 
strategies may have been selected. Finally, 
examining a combination of different test 
preparation products for the same student could 
provide in-depth identification of student profiles 
and their test preparation behaviors. 
Furthermore, qualitative studies could target 
some of the AOP profile members to recognize 
students’ response processes using probes, 
think-aloud, or focus group procedures. 
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Notes 


1. For more information about all ACT test prep products, visit https://www.act.org/content/act/en/products-and- 
services/the-act/test-preparation.html 

2. In 2014, the standard error of measurement of the SAT was approximately 32 points (College Board, 2015). This 
means that if a student retook the SAT with no additional learning, they have a 68% chance of scoring within 32 
points of their original test score. 

3. Effect size is based on the coefficient of 2.34, the standard deviation associated with individual tutoring hours 
(9.31), and the 2016 College Bound Seniors Report standard deviation of SAT total score (297.4). 

4. Software resets allow a student to clear responses for practice items and/or tests and take them again. 

5. ACT Composite score is the mean of the English, mathematics, reading, and science scores. 

6. Weused the Composite score rather than the subject tests both as an overall index of achievement on the 
subject tests and because it is typically the key marker of achievement for college admissions and scholarships. 

7. It was possible for a student to reset their practice tests and take practice tests multiple times. We used the most 
recent practice test score in our analysis. 

8. Fora detailed treatment of the LPA methodology see prior works (e.g., Geiser, 2012; Nylund, Asparouhov, & 
Muthén, 2007; Masyn, 2013; Morin, 2016; Muthén & Muthén, 1998-2015). For an overview of using LPA. 

9. Outliers were identified by running SAS “Outlier Macro” to calculate the interquartile range (IQR) to set up a 
“fence” outside of Q1 and Q3. Then, any values that fall outside of this fence are deleted since they considered 
outliers. 
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